Method and system for real-time continuous lane mapping and classification

ABSTRACT

The present disclosure discloses systems and method for real-time continuous lane mapping. The technique comprising receiving an input image of a road captured by an image sensor mounted on a vehicle; extracting one or more feature vectors from the image; extracting lane mark coefficients and lane type features from the one or more extracted feature vectors; detecting a lane mark by computing the coefficients and applying a pre-learned value; comparing the lane type relating to right and/or left lane markings with a predefined lane class; classifying the left and/or right lane markings based on the comparison and applying a pre-learned value; and generating a lane map along with the lane markings. The step of the lane detection and the classification is performed simultaneously based on the one or more extracted feature vectors.

CROSS-REFERENCE TO RELATED APPLICATION

This application claims priority to and the benefit of India PatentApplication No. 202011052696, filed on Dec. 3, 2020, the disclosure ofwhich is incorporated herein by reference in its entirety.

TECHNICAL FIELD

The present disclosure relates to field of autonomous driving. Moreparticularly, the present disclosure relates to lane detection andclassification for real-time continuous lane mapping.

BACKGROUND

Autonomous Driving Vehicles rely heavily on location of lanes forsuccessful control and navigation over roads. Of particular importanceto the successful control and navigation of the autonomous vehicles overroads is the ability to identify boundaries and area of traffic lanes.Apparently, among the complex and challenging tasks of such roadvehicles is road lane detection or road boundaries detection andclassification of lanes since driving constraints change with respect tothe class of the lane.

During the driving operation, humans use their optical vision forvehicle maneuvering, while autonomous vehicles use computer visiontechniques for their operations. Comparatively, it's easy for humans tofind the location of the lanes on roads whereas it is a difficult taskfor a computer vision system where lane detection is a crucial step indecision making while an autonomous vehicle operates. While trafficlanes are usually delineated by simple lines and patterns, it is oftendifficult in practice for autonomous vehicle driving systems to identifylane boundaries due to road deterioration in a quick time, lightingconditions, rain, and similarity with other objects and patterns thatmay be found in a traffic scene, such as other vehicles or road-sidestructures.

There are two major metrics in the evaluation for a lane detectionsystem, namely, Speed and Accuracy. Real-time decisions need to be madewith high accuracy for it to steer properly. Wrong lane detections canlead to fatal accidents in the real-world scenario. The existingtechniques disclose heuristics, which are followed by post-processingtechniques to identify lane segments. However, such heuristics andpost-processing techniques are not only expensive, but also fail toidentify lane segments when variations in road scene occurs. Also, theexisting solutions are not suitable for shorter range of FOV and lacksspeed and accuracy required for lane detection and classification.Further, the existing solutions may fail to identify lane segments ininclement weather conditions like rainy and snow scenarios where rainblobs or snow blobs obstruct the view of the lane.

Therefore, it is highly desirable to provide an efficient real-timecontinuous lane mapping technique with higher accuracy.

SUMMARY OF THE INVENTION

One or more shortcomings of the prior art are overcome, and additionaladvantages are provided by the present disclosure. Additional featuresand advantages are realized through the techniques of the presentdisclosure. Other embodiments and aspects of the disclosure aredescribed in detail herein and are considered a part of the disclosure.

It is to be understood that the aspects and embodiments of thedisclosure described above may be used in any combination with eachother. Several of the aspects and embodiments may be combined togetherto form a further embodiment of the disclosure.

In an aspect, the present disclosure provides a method for real-timecontinuous lane mapping for a host vehicle. The method comprising stepsof: receiving an input image of a road captured by an image capturingdevice mounted on the host vehicle; extracting one or more featurevectors from the image; extracting lane mark coefficients and lane typefeatures from the one or more extracted feature vectors; detecting alane mark by computing the coefficients and applying a pre-learnedvalue; comparing the lane type relating to right and/or left lanemarkings with a predefined lane class; classifying the left and/or rightlane markings based on the comparison and applying a pre-learned value;and generating a lane map along with the lane markings; wherein, thestep of the lane detection and the classification is performedsimultaneously based on the one or more extracted feature vectors.

In another aspect, the present disclosure provides a method, wherein theinput image may be a Ground Truth (GT) image generated from an originalimage using a lane labeler tool.

In another aspect, the present disclosure provides a method, wherein theinput image may be an RGB image that comprises scene covered in a FOV ofthe image sensor configured to capture front view from the host vehicle

In another aspect, the present disclosure provides a method, wherein thelane mark coefficients may comprise coefficients for representing acurvature of the road.

In another aspect, the present disclosure provides a method, wherein thelane type features may comprise a lane boundary, a starting position, adirection, grey-level intensity features, edge orientations, a shape, aposition of an object in the image, an aspect ratio that are implicitlylearned and modelled in a deep learning system.

In yet another aspect, the present disclosure provides a method, whereinthe lane mark may comprise a dotted lane type, a solid lane type, lanecolor and road characters.

In yet another aspect, the present disclosure provides a method, whereinthe extracting lane mark coefficients and lane type features may furthercomprise distinguishing a lane mark from objects present/drawn on theroad.

In another aspect, the present disclosure provides a method, whereingenerating a lane map along with lane markings may further compriseconcatenating steps of the lane detection and the classification; andplotting the lane map in real time on a display.

In an aspect, the present disclosure provides a system for real-timecontinuous lane mapping for a host vehicle. The system comprises animage sensor configured to capture an image of a road; and a controllercommunicatively connected to the image sensor and configured to: receivean input image of a road captured by the image capturing device; extractone or more feature vectors from the image; extract lane markcoefficients and lane type features from the one or more extractedfeature vectors; detect a lane mark by computing the coefficients andapplying a pre-learned value; compare the lane type relating to rightand/or left lane markings with a predefined lane class; classify theleft and/or right lane markings based on the comparison; and generate alane map along with the lane markings; wherein, the controller isconfigured to simultaneously detect and classify the lane based on theone or more extracted feature vectors.

In another aspect, the present disclosure provides a system, wherein thecontroller may be further configured to detect the lane markcoefficients comprising coefficients for representing a curvature of theroad.

In yet another aspect, the present disclosure provides a system, whereinthe controller may be further configured to detect a lane boundary, astarting position, a direction, grey-level intensity features from thelane type features, edge orientations, a shape, a position of an objectin the image, an aspect ratio that are implicitly learned and modelledin a deep learning system.

In another aspect, the present disclosure provides a system, wherein thecontroller may be further configured to identify the lane mark based ona dotted lane type, a solid lane type, a lane color and road characters.

In another aspect, the present disclosure provides a system, wherein thecontroller may be further configured to concatenate the lane detectionand the classification to generate the lane map along with lanemarkings, and plot the lane map along with the lane markings in realtime.

In an aspect, the present disclosure provides a non-transitorycomputer-readable medium. The medium comprising computer-readableinstructions for real-time continuous lane mapping for a host vehicle,when executed by a host vehicle, causes a processor to: receive an inputimage of a road captured by an image sensor mounted on the host vehicle;extract one or more feature vectors from the image; extract lane markcoefficients and lane type features from the one or more extractedfeature vectors; detect a lane mark by computing the coefficients andapplying a pre-learned value; compare the lane type relating to rightand/or left lane markings with a predefined lane class; classify theleft and/or right lane markings based on the comparison; and generate alane map along with the lane markings; wherein, the lane detection andthe classification is performed simultaneously based on the one or moreextracted feature vectors.

In another aspect, the present disclosure provides a computer-readablemedium, which may further comprise instructions that cause the processorto detect the lane mark coefficients further comprising coefficients forrepresenting a curvature of the road.

In another aspect, the present disclosure provides a computer-readablemedium, which may further comprise instructions that cause the processorto detect a lane boundary, a starting position, a direction, andgrey-level intensity features from the lane type features from the lanetype features, edge orientations, a shape, a position of an object inthe image, an aspect ratio that are implicitly learned and modelled in adeep learning system.

In another aspect, the present disclosure provides a computer-readablemedium, which may further comprise instructions that cause the processorto identify lane mark based on a dotted lane type, a solid lane type, alane color and road characters.

In yet another aspect, the present disclosure provides acomputer-readable medium, which may further comprise instructions thatcause the processor to concatenate the lane detection and theclassification to generate the lane map along with lane markings.

In yet another aspect, the present disclosure provides acomputer-readable medium, which may further comprise instructions thatcause the processor to plot the lane map along with the lane markings inreal time.

BRIEF DESCRIPTION OF DRAWINGS

The accompanying drawings, which are incorporated in and constitute apart of this disclosure, illustrate exemplary embodiments and, togetherwith the description, serve to explain the disclosed principles. In thefigures, the left-most digit(s) of a reference number identifies thefigure in which the reference number first appears. The same numbers areused throughout the figures to reference like features and components.Some embodiments of system and/or methods in accordance with embodimentsof the present subject matter are now described, by way of example only,and with reference to the accompanying figures, in which:

FIG. 1 illustrates blind spot for conventional camera in existingcomputer visions systems of autonomous vehicles as compared to a cameradisclosed in the present disclosure.

FIG. 2 illustrates lane detection using the camera according to anaspect of the present disclosure.

FIG. 3 shows an exemplary architecture for detecting lane pattern inaccordance with an aspect of the present disclosure.

FIG. 4 shows a detailed block diagram of the real time continuous lanemapping system in accordance with an aspect of the present disclosure.

FIG. 5 illustrates a multi-step process to achieve real-time continuouslane mapping according to an aspect of the present disclosure.

FIG. 6 illustrates a block diagram of dataset preparation according toan aspect of the present disclosure.

FIG. 7 illustrates an overview of model architecture according to anaspect of the present disclosure.

FIG. 8 illustrates an architecture of a shared encoder according to anaspect of the present disclosure.

FIG. 9 illustrates a detailed model architecture according to an aspectof the present disclosure.

FIG. 10 illustrates a training process of complete architectureaccording to an aspect of the present disclosure.

FIG. 11 illustrates a training process of a regressor of thearchitecture according to an aspect of the present disclosure.

FIG. 12 illustrates a training process of a classifier of thearchitecture according to an aspect of the present disclosure.

FIG. 13 illustrates an inference process of the architecture accordingto an aspect of the present disclosure.

FIGS. 14a-c illustrates different scenarios for handled by the real-timecontinuous lane mapping system according to an aspect of the presentdisclosure.

FIGS. 15a-c illustrates lane maps produced in different challengingcases illustrated in FIG. 14a-c by the real-time continuous lane mappingsystem according to an aspect of the present disclosure.

DETAILED DESCRIPTION

Referring in the present document, the word “exemplary” is used hereinto mean “serving as an example, instance, or illustration.” Anyembodiment or implementation of the present subject matter describedherein as “exemplary” is not necessarily to be construed as preferred oradvantageous over other embodiments.

While the disclosure is susceptible to various modifications andalternative forms, specific embodiment thereof has been shown by way ofexample in the drawings and will be described in detail below. It shouldbe understood, however that it is not intended to limit the disclosureto the particular forms disclosed, but on the contrary, the disclosureis to cover all modifications, equivalents, and alternatives fallingwithin the scope of the disclosure.

The terms “comprises”, “comprising”, or any other variations thereof,are intended to cover a non-exclusive inclusion, such that a setup,device that comprises a list of components does not include only thosecomponents but may include other components not expressly listed orinherent to such setup or device. In other words, one or more elementsin a system or apparatus proceeded by “comprises . . . a” does not,without more constraints, preclude the existence of other elements oradditional elements in the system or apparatus or device. It could benoted with respect to the present disclosure that the terms like “asystem for real-time continuous lane mapping”, “the system” areinterchangeably used throughout the description and refer to the samesystem. Similarly, terms like “Autonomous Driving Vehicles”, “autonomousvehicles”, are interchangeably used throughout the description.

Disclosed herein are the techniques for real-time continuous lanemapping. In an exemplary embodiment of the present disclosure, an imagecapturing device is mounted on a vehicle, preferably at bumper of thevehicle, but not limited thereto. As an example, the image capturingdevice may include, but not limited to a fisheye camera which provides awider Field of View (FOV). The fish-eye camera continuously captures animage of a road and sends these images for further processing. Forexample, the images captured by the camera is provided to a lanedetection and a classification system. The system extracts requiredfeatures from the image and performs the lane detection and theclassification simultaneously based on the extracted features.

This achieves advantages with respect to an accuracy and a speed andcontributes to efficient and robust real time lane mapping. The presentdisclosure achieves these advantage(s) in a manner as described belowwith respect to the drawings.

FIG. 1 illustrates a blind spot for conventional camera as compared to afish-eye camera used in the present disclosure. As already indicated inthe background section, the conventional camera has a higher Field ofView (FOV) i.e., —6-100 meters restricting it to perceive—6 metersdirectly ahead of the vehicle which is crucial for any lane detectionsystem. The area not perceivable by conventional cameras is known as theblind spot. The fisheye camera is able to avoid this problem as it canperceive the area in the blind spot making it favorable for real timelane mapping systems. The Fisheye camera can be used in both cases wherelower FOV is required and also where higher FOV is required thateliminates the use of multiple cameras for different cases. Although,the fish-eye camera is preferable for image capturing in the presentdisclosure, however, it may be noted that any other such camera orsensing device that fulfills desired requirement of the presentdisclosure may be used for image capturing.

FIG. 2 illustrates real time lane mapping using fisheye camera accordingto an embodiment of the present disclosure. The present disclosureprovides a computationally efficient and optimized design for detectingand classifying the lanes using a fish-eye camera's frontal view invehicles. The fish-eye camera captures images with a length of view upto 6 meters. The system predicts three coefficients namely a, b, c and nclass probabilities in an end-to-end model. The end-to-end modelspecifies that the lane detection and the classification are performedsimultaneously and thus avoids a post processing. This makes the wholesystem computationally efficient with an optimized design.

The present disclosure uses Deep Learning based approach that makes theinference faster with respect to other conventional computer visionapproaches. A coefficient based lane detection method requires no postprocessing and can be directly used for decision making in autonomousvehicles due to its mathematical equation-like output. The whole processis end-to-end and thus provides results in real time. Using threecoefficients for each lane totaling to six coefficients the system formsthe quadratic curve. Then, the equation and the classes are used to plotthe output on the image. The lane detection algorithm uses the fish eyeimages to predict a parabolic equation defined as: ax²+bx+c=y and theconsecutive classes for both of the lanes. The system is able toclassify both the lanes by the camera into various classes. Forexample—a solid lane, a dotted lane, etc.

FIG. 3 shows an exemplary architecture for detecting and aclassification of a lane in accordance with some embodiments of thepresent disclosure.

The architecture 300 comprises a vehicle 301, a real time continuouslane mapping system 302 and a display device 308. As an example, thevehicle 301 may be a car, a truck, a bus, and the like. Input imagescaptured by an image capturing device 304 configured on the vehicle 301and provided to the real time continuous lane mapping system 302. As anexample, the image capturing device 304 configured on the vehicle 301,may access an image repository or a computing device such as a mobile, adesktop, a laptop and the like associated with the image capturingdevice. As an example, the image capturing devices may include, but notlimited to, a camera. In some embodiments, one or more image capturingdevices may be configured at different positions on the vehicle 301. Thepreferred position is at the bumper of the device. The real timecontinuous lane mapping system 302 may be hosted on a server. In someembodiments, the server in which the real time continuous lane mappingsystem 302 is hosted may be a local server configured in the vehicle 301as shown in the FIG. 3. In some other embodiments, the server in whichthe real time continuous lane mapping system 302 is hosted may be aremote server or a cloud server.

Further, the real time continuous lane mapping system 302 may include acontroller 305, an Input/Output (I/O) interface 303 and a memory 306.The I/O interface 303 may receive an input image/training image from adata source among the one or more data sources. In some embodiments, theinput image may be captured by the image capturing device 304 configuredto capture a front view from the vehicle 301. In some embodiments, thetraining image may be a Ground Truth (GT) image comprising one or morelane markings and co-ordinates of the one or more lane markings. Theinput image received through the I/O interface 303 may be stored in thememory 306. Further, the I/O interface 303 may access a historical lanedata stored in the database 307 associated with real time continuouslane mapping system 302. As an example, the historical lane data mayinclude, but not limited to, lane patterns detected from previous imagesof lanes captured in real-time by the image capturing device. In someembodiments, the database 307 may further include, but not limited to,training images of the lanes captured in different weather conditionsand light conditions, and other related image parameters. Further,controller 305 may extract all of the features or feature maps from theinput image. These feature maps are flattened to form a feature vector.The controller 305 extracts lane mark coefficients and lane typefeatures from the extracted features. The controller further detects alane mark by computing the coefficients and applying a pre-learnedvalue, compares the lane type relating to right and/or left lanemarkings with a predefined lane class. Based upon this comparison, thecontroller 305 classifies left and/or right lane markings and applying apre-learned value stored in the memory 306 and generates a lane mapalong with lane markings. It may be worth noted that the classificationand the detection of lanes is performed simultaneously by the controller305 based on the historical lane data, using a trained machine learningmodel.

As an example, the trained machine learning model may detect the lanepattern under various conditions such as noisy conditions occurring dueto a presence of dust/water on the image capturing device, due to rainand the like, varying illumination conditions due to shadows ofsurrounding objects, tunnels, weather conditions and the like. Further,in some embodiments, the detected lane pattern may be displayed usingthe display device 308 associated with the system 302. In someembodiments, the detected lane pattern may be displayed on an originalimage, from which the GT image was generated. As an example, thedetected lane pattern is a solid lane. The controller 305 may display aphrase “solid lane” on the corresponding lane in the original image.

FIG. 4 shows a detailed block diagram of the real time continuous lanemapping system in accordance with some embodiments of the presentdisclosure.

In some implementations, the real time continuous lane mapping system400 may include data and modules 407. As an example, the data may bestored in a memory 404 configured in the real time continuous lanemapping system 400. In one embodiment, the data may include an inputimage data 403, a processed image data 404, a lane pattern data 405 andother data 406.

In some embodiments, the data may be stored in the memory 404 in form ofvarious data structures. Additionally, the data can be organized usingdata models, such as relational or hierarchical data models. The otherdata 406 may store data, including a temporary data and temporary files,generated by the modules 407 for performing the various functions of thereal time continuous lane mapping.

In some embodiments, the data stored in the memory may be processed bythe modules 407 of the real time continuous lane mapping system 400. Themodules 407 may be stored within the memory 404. In an example, themodules 407 communicatively coupled to the controller 401 configured inthe real time continuous lane mapping system 400, may also be presentoutside the memory 404 as shown in FIG. 4 and implemented as hardware.As used herein, the term modules 407 may refer to an applicationspecific integrated circuit (ASIC), an electronic circuit, a processor(shared, dedicated, or group), a controller and memory that execute oneor more software or firmware programs, a combinational logic circuit,and/or other suitable components that provide the describedfunctionality.

In some embodiments, the modules 407 may include, for example, a encodermodule 408, a regressor module 409, a classifier module 410, aconcatenation module 412 and other modules 411. The other modules 411may be used to perform various miscellaneous functionalities of the realtime continuous lane mapping system 400. It will be appreciated thatsuch aforementioned modules 407 may be represented as a single module ora combination of different modules.

In some embodiments, the encoder module 408 may receive an input imagefrom the image capturing device. The input image thus received may bestored as the input image data 403.

In some embodiments, the input images captured by the image capturingdevice configured on the vehicle 301 are provided to the real timecontinuous lane mapping system 400. As an example, input images may beprovided by, for example, the image capturing device configured on thevehicle 301.

As previously discussed, in some embodiments, the training image is aGround Truth (GT) image comprising one or more lane markings andco-ordinates of the one or more lane markings. The GT image may begenerated from the input image captured by the image capturing deviceusing any available lane labeler tool. In some embodiments, the inputimage may be an RGB image that may include scene covered in the FOV ofthe image capturing device configured to capture the front view from thevehicle 301. The different modules/units are now described in detail inFIGS. 5-15.

FIG. 5 illustrates a complete multi-step process 500 to achieve areal-time continuous lane mapping according to an embodiment of thepresent disclosure. The various steps involved are namely, a data setpreparation 501, a model architecture 502, a training phase 503 and aninference 504. Each of these steps/blocks are discussed in laterdrawings.

FIG. 6 illustrates a block diagram of dataset preparation 600 (501 ofFIG. 4) according to an embodiment of the present disclosure. In anexemplified embodiment of the present disclosure, the datasetpreparation is done using the fisheye camera lens installed on thevehicle. After capturing the images, lane markings and type of lanemarkings is done using a Lane Labeler tool. The left and right lanes areaccordingly marked by different colors in a marked image as shown inFIG. 6. Thus, a GT image is obtained.

FIG. 7 illustrates an overview of model architecture 700 (502 of FIG. 4)according to an embodiment of the present disclosure. In an exemplaryembodiment, the model architecture 700 consists of three majorcomponents namely, a shared encoder 701, a regressor 702 and aclassifier 703. The first layer of the model architecture is a LambdaLayer which allows for variable size input image. The architecture runson a 224×224 resolution image. The shared encoder 701 is a convolutionalneural network model, which extracts the feature maps from the image.The feature maps are flattened to form a feature vector. The featurevector from the shared encoder 701 is then passed to the regressor (or aregressor unit) 702 and the classifier (or a classifier unit) 703 tomake coefficient and class predictions. the regressor 702 is used topredict the coefficients which are continuous variables and theclassifier 703 predicts class probabilities which are discreet variablesrelated to lane. A Leaky Rectified Linear Unit (ReLU) activation layeris used as a non-linearity function aided in improving the results.Leaky ReLU function is an upgrade from ReLU which is able to calculategradients in the negative region of the graph. The shared encoder 701has various layers connected in serial. The shared encoder 701 extractsfeature from the image which are common to both the classifier 702 andthe regressor 703.

FIG. 8 illustrates an architecture of the shared encoder 800 accordingto an embodiment of the present disclosure. The shared encoder is usedfor extraction of features information such as texture, color, edgesetc. that is essential for the detection and the classification of lanesand shared for both operations. In the architecture of shared encoder, Bis a batch size, H is a height of the image and W is a width of theimage. The architecture of shared encoder consists of following blocks:

Data Layer: Data layer is used to interface the data with the modelarchitecture. It consist of two layers, an input layer which receivesthe data (i.e. Image) and stored it to memory and a Lambda layer whichresizes the image of any size to (B×224×224×3) which is required by amodel.Batch Normalization: This layer is used to normalize the data so thatthe effect of covariate shift can be reduced.Cony Block: This block is responsible for learning the features. Itconsists of three sub-blocks. A zero padding is used for adding extrapixels to the boundary of image so that the size of a input and a outputto a convolution would be same. A convolution block contains a learnableparameter and their value is adjusted during training and used toextract features.Leaky Relu: Leaky Relu is used as non-linear activation function whichadds non-linearity to otherwise linear blocks of the architecture. Thuscony block is able to learn complex features.Max pooling block: This layer is used to retrieving contextualinformation from the intermediate feature maps by taking out maximumvalues and reduces the size of feature maps thus complexity of themodel.Flatten Layer: This layer is used to make 2D feature maps to a 1D vectorwhich makes it compatible to be passed in feedforward neural network.

FIG. 9 illustrates a detailed model architecture 900 according to anembodiment of the present disclosure. This architecture provides adetailed view of a regressor unit 901 and a classifier unit 902. In anexemplary embodiment, after extracting features from the shared encoder,this information is used to detect and classify the lanes. The regressorunit 901 is responsible for detecting the position of lanes by giving 6coefficients, 3 for each lane and the classifier unit 902 classifies thelanes whether solid, dotted etc. The model architecture performssimultaneous detection and classification of lanes.

Regressor 901:

The main function of this block is to learn the transformation offeatures containing an information about the position of lanes tomathematical quantities which are coefficients. According to anexemplary aspect of the present disclosure, lanes are modeled, forexample, as second-degree polynomial curve (a*x*x+b*x+c) where a, b, care the coefficients. As this transformation requires lots ofcomputation, a Dense layer is used, which is made up of neurons, whichreceives inputs and calculates output as {y=f(W*x+B)}, wherein:x—represents an Input vectory—represents an Output vectorW, B—represents Weights and bias (these are the parameters which themodel learns during training phase)f( )—represents a Non-linear function Leaky Relu (to add non-linearityto linear computation so that the regressor can learn non-lineartransformation)

In addition to above, a dropout layer is used which is a sort of aregularization that is used to stop overfitting. Finally, a last denselayer gives contains 6 neurons nodes to provide coefficients.

Classifier 902:

The main function of this block is to learn the classification on thebasis of features extracted by shared encoder. The other layers used areDense layer, Dropout layer as explained in the regressor used to convertfeatures to a four-dimensional feature vector which represents the classof lanes. The number of nodes in the classifier is less than theregressor as the classification requires a less transformation ascompared to the regression which extract an exact mathematical quantity.The classifier layer uses SoftMax non-linear activation function whichgives joint probabilities for 4 output nodes and one with the maximumvalue taken as 1 and other are 0 and lanes are classified as:

[1,0,0,0]—Solid, Solid

[0,1,0,0]—Solid, Dotted

[0,0,1,0]—Dotted, Solid

[0,0,0,1]—Dotted, Dotted

Further, a concatenation layer 903 is used to combine the result of boththe regressor and the classifier.

FIG. 10 illustrates a training process 1000 of a whole architectureaccording to an embodiment of the present disclosure. In an exemplaryembodiment, first of all, the complete architecture got trained on arespective ground truth so that parameters of the shared encoder gotadjusted and it is able to extract the useful features for the detectionand the classification tasks. The architecture is trained with all theshared encoder, the regressor and the classifier part trainable with acustom loss which comprises a combination of Mean square error and acategorical cross entropy loss. After training the whole architecture,the next step is to finetune the regression branch and the classifierbranch separately.

FIG. 11 illustrates a training process of a regressor of thearchitecture according to an embodiment of the present disclosure. Inthis process, the regression branch is finetuned. In this step, theshared encoder and the classifier part is freezed. Only regressionbranch is trainable which is trained by the categorical cross entropyloss using ADAM optimizer. In an exemplary embodiment, ⅓^(rd) crop ofthe image is used to provide less weightage to the upper part of theimage (usually sky part). This provides improved results as less data isrequired as compared to full image.

FIG. 12 illustrates a training process of a classifier of thearchitecture according to an embodiment of the present disclosure. Afterthat, the classifier branch is finetuned and the shared encoder and theregressor got freezed. The classifier is trained using a categoricalcross entropy loss with ADAM optimizer. The model is trained withlearning rate of le-3.

Training process as shown in FIGS. 11 and 12 comprises of passing theinput samples to the architecture to which it output its prediction,which are 6 coefficients for the lane detection and 4 classificationcoefficients. Then these respected predictions get compared with groundtruth via a loss function to measure how accurate the predictions and onthe basis of an error which is back propagated so that the model canadjust its parameters for a better prediction next time.

For Regressor:

The Loss function used for the regressor is a mean square error loss andits ground truth comprises of the exact value of normalized 6coefficients of polynomial modeled lanes. ADAM optimizer is used duringtraining.

For Classifier:

The Loss function used for the classifier is the categorical crossentropy loss and ground truth comprises of the exact value of 4 binaryvalues which represents classes in terms of one hot encoding. e.g.[1,0,0,0] for solid, solid lanes. ADAM optimizer is used duringtraining.

FIG. 13 illustrates an inference process of the architecture accordingto an embodiment of the present disclosure. During the inference, losslayers are removed, and the image is fed to the shared encoder. Theoutputs from the classifier and the regressor are concatenated. The laneis plotted with the lane and lane types using the Image Plotter.

FIG. 14a-c illustrates different scenarios handled by the real-timecontinuous lane mapping system according to an embodiment of the presentdisclosure. For example, FIG. 14a represents an image taken by thecamera wherein the road/lanes have different characters drawn on them.FIG. 14b represents an image of the road/lane taken on a rainy day. FIG.14c represents an image of the road/lane taken inside a tunnel when thelane is not much visible.

FIGS. 15a-c illustrates lane maps produced in different scenariosillustrated in FIGS. 14a-c by the real-time continuous lane mappingsystem according to an embodiment of the present disclosure. Thus, thepresent disclosure shows improved results on various scenarios includingbut not limited to, straight lane, curved lane, tunnel case, change inillumination case, rain images, road characters etc.

The foregoing description of the various embodiments is provided toenable any person skilled in the art to make or use the presentdisclosure. Various modifications to these embodiments will be readilyapparent to those skilled in the art, and the generic principles definedherein may be applied to other embodiments without departing from thespirit or scope of the disclosure. Thus, the present disclosure is notintended to be limited to the embodiments shown herein, and instead theclaims should be accorded the widest scope consistent with theprinciples and novel features disclosed herein.

While the disclosure has been described with reference to a preferredembodiment, it is apparent that variations and modifications will occurwithout departing the spirit and scope of the disclosure. It istherefore contemplated that the present disclosure covers any and allmodifications, variations or equivalents that fall within the scope ofthe basic underlying principles disclosed above.

We claim:
 1. A method for real-time continuous lane mapping for a hostvehicle, the method comprising steps of: receiving an input image of aroad captured by an image sensor mounted on the host vehicle; extractingone or more feature vectors from the input image; extracting lane markcoefficients and lane type features from the one or more extractedfeature vectors; detecting a lane mark by computing the coefficients andapplying a pre-learned value; comparing the lane type relating to rightand/or left lane markings with a predefined lane class; classifying theleft and/or right lane markings based on the comparison and applying apre-learned value; and generating a lane map along with the lanemarkings; wherein, the step of the lane detection and the classificationis performed simultaneously based on the one or more extracted featurevectors.
 2. The method of claim 1, wherein the input image is a GroundTruth (GT) image generated from an original image using a lane labelertool.
 3. The method as claimed in claim 1, wherein the input image is anRGB image that comprises a scene covered in a FOV of the image sensorconfigured to capture front view from the host vehicle.
 4. The method ofclaim 1, wherein the lane mark coefficients comprise coefficients forrepresenting a curvature of the road.
 5. The method of claim 1, whereinthe lane type features comprise a lane boundary, a starting position, adirection, grey-level intensity features, edge orientations, a shape, aposition of an object in the image, an aspect ratio that are implicitlylearned and modelled in a deep learning system.
 6. The method of claim1, wherein the lane mark comprises a dotted lane type, a solid lanetype, a lane color and road characters.
 7. The method of claim 1,wherein the extracting lane mark coefficients and lane type featuresfurther comprise distinguishing a lane mark from objects present/drawnon the road.
 8. The method of claim 1, wherein the generating a lane mapalong with lane markings further comprises: concatenating steps of thelane detection and the classification; and plotting the lane map in realtime on a display.
 9. A system for real-time continuous lane mapping fora host vehicle, comprises: an image sensor configured to capture animage of a road; and a controller communicatively connected to the imagesensor and configured to: receive an input image of a road captured bythe image sensor; extract one or more feature vectors from the image;extract lane mark coefficients and lane type features from the one ormore extracted feature vectors; detect a lane mark by computing thecoefficients and applying a pre-learned value; compare the lane typerelating to right and/or left lane markings with a predefined laneclass; and classify the left and/or right lane markings based on thecomparison and apply a pre-learned value; and generate a lane map alongwith the lane markings; wherein, the controller is configured tosimultaneously detect and classify the lane based on the one or moreextracted feature vectors.
 10. The system of claim 9, wherein thecontroller is further configured to detect the lane mark coefficientscomprising coefficients for representing a curvature of the road. 11.The system of claim 9, wherein the controller is further configured todetect a lane boundary, a starting position, a direction, grey-levelintensity features from the lane type features, edge orientations, ashape, a position of an object in the image, an aspect ratio that areimplicitly learned and modelled in a deep learning system.
 12. Thesystem of claim 9, wherein the controller is further configured toidentify the lane mark based on a dotted lane type, a solid lane type, alane color and road characters.
 13. The system of claim 9, wherein thecontroller is further configured to: concatenate the lane detection andthe classification to generate the lane map along with lane markings;and plot the lane map along with the lane markings in real time.
 14. Anon-transitory computer-readable medium comprising computer-readableinstructions for real-time continuous lane mapping for a host vehicle,when executed by a processor, causes the processor to: receive an inputimage of a road captured by an image sensor mounted on the host vehicle;extract one or more feature vectors from the image; extract lane markcoefficients and lane type features from the one or more extractedfeature vectors; detect a lane mark by computing the coefficients andapplying a pre-learned value; compare the lane type relating to rightand/or left lane markings with a predefined lane class; classify theleft and/or right lane markings based on the comparison; and generate alane map along with the lane markings; wherein, the lane detection andthe classification is performed simultaneously based on the one or moreextracted feature vectors.
 15. The computer-readable medium of claim 14,further comprising instructions that cause the processor to detect thelane mark coefficients further comprising coefficients for representinga curvature of the lane on the road.
 16. The computer-readable medium ofclaim 14, further comprising instructions that cause the processor todetect a lane boundary, a starting position, a direction, and grey-levelintensity features from the lane type features from the lane typefeatures, edge orientations, a shape, a position of an object in theimage, an aspect ratio that are implicitly learned and modelled in adeep learning system.
 17. The computer-readable medium of claim 14,further comprising instructions that cause the processor to identify thelane mark based on a dotted lane type, a solid lane type, a lane colorand road characters.
 18. The computer-readable medium of claim 14,further comprising instructions that cause the processor to concatenatethe lane detection and the classification to generate the lane map alongwith lane markings.
 19. The computer-readable medium of claim 14,further comprising instructions that cause the processor to plot thelane map along with the lane markings in real time.